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This paper discusses an idealization of the situation in which it is 
required to send information over two separate channels, as in a 
packet communication network, and it is desired to recover as much 
as possible of the original information should one of the channels 
break down. Let {Xk} *=i be a sequence of independent copies of the 
binary random variable X, where Pr[X = 0} = Pr{X = 1} = V4. 
Assume that this sequence appears at a rate of one symbol per second 
as the output of a data source. An encoder observes this sequence 
and emits two binary sequences at rates R\, R 2 ^ 1. These sequences 
are such that, by observing either one, a decoder can recover a good 
approximation to the source output and, by observing both sequences, 
a decoder can obtain a better approximation to the source output. 
Letting D\ , D2 , Do be the error rates that result when the streams at 
rate R\, rate R2, and both streams are used by a decoder, respectively, 
our problem is to determine (in the usual Shannon sense) the set of 
achievable quintuples (Ri, R 2 , Do, Di, D 2 ). Our main result is a 
"converse" theorem that gives a necessary condition on the achievable 
quintuples. 

I. INTRODUCTION 

Let {Xk}k=i be a sequence of independent copies of the binary 
random variable X, where Pr{X = 0} = Pr(X = 1} = V2. Assume that 
this sequence appears at a rate of one symbol per second as the output 
of the data source in Fig. 1. The encoder in the figure observes this 
sequence and emits two binary sequences at rates R i} R2 ^ 1. These 
sequences are such that, by observing either one, a decoder can recover 
a good approximation to the source output, and by observing both 
sequences a decoder can obtain a better approximation to the source 
output. Letting D\, D 2 , Do be the error rates which result when the 
streams at rate Ri, rate R 2 , and both streams are used by a decoder, 
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Fig. 1 — Source encoder-decoder. 

respectively, our problem is to determine (in the usual Shannon sense) 
the set of achievable (Ri, R 2 , Do, D\, D 2 ). 

To fix ideas, let us say that Ri = R 2 = V2, D = and A = D 2 . Thus 
the source sequence at rate 1 is to be encoded into two sequences of 
rate Vi each, such that the original sequence can be recovered from 
these two encoded sequences with approximately zero error rate (i.e., 
Do = 0). Our question then becomes: How well can we reconstruct the 
source sequence from one of the encoded streams? (That is, what is D\ 
= D 2 ?) A simple-minded approach would be to let the encoded streams 
consist of alternate source symbols, which will allow D = 0. In this 
case, D\ = D 2 = l A, since by observing every other source symbol a 
decoder will make an error half the time on the missing symbol. Is it 
possible to do better? El Gamal and Cover 1 have looked at this problem 
and have a theorem that can be used to show that we can make D\ = 
Z>2 = (V2 — l)/2 ~ 0.207. In the present paper, we prove a theorem 
from which it follows that (with R 1 = R 2 = % D = 0),Di = D 2 > Ve. An 
exact determination of the best Di = D 2 is at present an open problem.* 

Let us remark that this problem can be generalized in an obvious 
way to an arbitrary source {Xk } , and arbitrary distortion measure. An 
especially interesting case is where the {Xk} are Gaussian and the 
distortion is the squared-error criterion. For this case, L. H. Ozarow 3 
has obtained the complete solution. 

II. FORMAL STATEMENT OF PROBLEM AND RESULTS 

Let 38 = {0, 1} , and let d//(x, y), x, y e 38 N , be the Hamming distance 
between the binary iV-vectors x, y, i.e., d/j(x, y) is the number of 
positions in which x and y do not agree. A code with parameters (N, 
Mi , M 2 , Do, D\ , D 2 ) is a quintuple of mappings (f\,f 2 ,go,g\,g 2 ) where 



* In Ref. 2, Witsenhausen proved a closely related result which encourages the 
conjecture that D\ = D2 = 0.207 is in fact the best possible. 
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f a :a N -*{l,-",M a ), a =1,2 (la) 

£ a :{l,2, ...,M a }->#", a = 1,2 (lb) 

£ : {1, 2, • • ., My) x {1, 2, • • -, M 2 ) -> **. (lc) 

The source output is a random vector X uniformly distributed on 
& N . Define 



£„ = £„• /.(X), a = 1, 2, 


(2a) 


and 




£o = £o[A(X),/ 2 (X)]. 


(2b) 


Then the average error -rates are 




D a = ^Ed H (X,X»), a = 1,2 


(3a) 


Do = lzEdH(X,Xo). 


(3b) 



N 

We say that a quintuple (f?i, i? 2 , do, d\, d 2 ) is achievable, if, for 
arbitrary e > 0, there exists, for N sufficiently large, a code with 
parameters 

(N, M u M 2 , D , D u D 2 ), 

where M a < 2 <A+€)iV , a = 1, 2, and Z)„ < d a + e, a = 0, 1, 2. The 
relationship of this formalism to the system of Fig. 1 should be clear. 
Our problem is the determination of the set of achievable quintuples, 
and our main result is the following "converse" theorem. 
Theorem: If(Ri, R 2 , do, d\, d 2 ) is achievable, then 

p l p > / 2 " h{do) ~ h ^ + 2d2) u\ 

«l + «2 > | 2 _ ft(do) _ A(2A + da)> (4) 

where 

[0, A = 0, 

/i(A) = \ -A log 2 A - (1 - A) log 2 (1 - A), < A < V 2 , 

[l, A>y 2 . 

All logarithms in this paper are taken to the base 2. 

Discussion: When Ui = R2 = %, do = 0, and d\ = d 2 , the rate- 
distortion bound implies that 1 - h(d x ) < Ri = x h, or d\ > 0.11. Our 
result (4), however, yields h($d\) > 1, or d\ > Vs. 

III. INFORMAL OUTLINE OF THE PROOF OF THE THEOREM 

In this section, we give an informal "e-free" discussion which con- 
tains the main ideas behind the proof of our theorem. 
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Let (fi, f2,go, gi,g2)be a code with parameters (N, Mi, M2, Do, Di, 
D 2 ), where M a « 2 RaN (a = 1, 2) and D a s d a (a = 0, 1, 2). Let 

At = {x e #": /i(x) = *'}> 1 < j < Mi, (5a) 

*/-{»€#*: A(x)-/}, 1<><M 2 , (5b) 

Cy = A,- n Bj. (5c) 

For this informal discussion, let us assume that for all x e SS N , 

d a *-d„(x,g a of a (x)), a =1,2 (6a) 

do ~ i rf«(x, 0,(/i(x) ( / 2 (x))). (6b) 

Also, since ^^\ A, = ^ N , the average cardinality of the At is 2 N /Mi st 
2 m ~ Ri) . For our informal discussion, we will assume that all the A, 
have approximately equal cardinality, so that 

\Ai\a2 m -**\ l<i<M. (7) 

where | A | denotes the cardinality of the set A. 
Now for 1 <y < M2, define 

#,= S A*. (8) 

In other words, i£, is the set of x e 3S N such that, for some x' e $ N , fi (x) 
= fi (x' ) and f 2 (x' ) = j. The proof proceeds by combining two bounds 
on \Kj\. 
We begin by obtaining an upper bound on \Kj\, Let 

A,= max d H (xi,x 2 ) 

x,,x 2 tK, 

be the diameter of Kj . Now the subset of &S N with diameter A with 
largest cardinality is a sphere with radius A/2. 
Thus 

\Kj\< £ ( ^ I < 2 Nm ' /m , 

or 

ilog|X,|<A(A y /2iV). (9) 

Next let Xi, x 2 , e A"; achieve <£h(xi, x 2 ) = Ay. From the definition of Kj 
(8), a pair x'i , x 2 exists such that 

/• 1 (x 1 )=/".(x' 1 ), /!(*)- /i<*i) (10a) 
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and 

f2(x\)=Mx'2)=j. (10b) 

Now (10a), and (6a) with a = 1, imply that 

rf w (x,, x'i) < 2rf,N, cMx 2 , xo) < 2diN, (11a) 

and (10b) and (6a) with a = 2 imply that 

d H (x' i ,x 2 )^2d 2 N. (lib) 

The triangle inequality and inequalities (11) yield 

A y = rf w (xi, x 2 ) < rf/r(xi, x 2 ) + d«(xi, xi) 

+ d„(x 2 ,x 2 )<2d 2 N + 4d i N 

or 

-2<d 2 + 2d 1 . (12) 

2iV 

Inequalities (12) and (9) yield, for 1 <y < Af 2 , 
ilog|«/|<A(cfe + 2d 1 ), 

and, averaging over 7, we have 

1 ^2 

- S Pr{/ 2 (X) =>} log I Kj \ < /i(d 2 + 2d,). (13) 

iV y=l 

This is the first of our bounds on \Kj\. 

We now obtain a lower bound on \Kj\. For 1 <y < Af 2 , let m, be the 
number of i(l < i < Mi) such that C,> ^ <£. Then, using (7) and (8), 

log \Kj\ss log mj + N(l - i?, ). (14) 

Further, from the rate-distortion bound (since X can be constructed 
from fi(X) and f 2 (X) with an average distortion d ), 

[1 -Mdo)]iV</(/l(X), f 2 (X);X) 

<#(/., f2)=H(f 2 ) + H(fi\f 2 ) 

< logM 2 + H(fi I f 2 ) = log M 2 + £ Fr{f 2 (X) =j)H(f i \ f 2 =j). (15) 

Now, given that / 2 (X) =>, f\ (X) takes values in a set of cardinality m> ». 
Thus, H( f\\f 2 =j) can be overbounded by log mj. Since M 2 a 2 NR * , we 
have, from (15), 

JS t PHf-2 =j) log mj > [1 - Wo)] - i? 2 , 

iVy=l 
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and, from (14), 

i 2 Pr{/ 2 ~j)\og\Kj\ > 2 - h(do) -Ri- R 2 . (16) 

This is our second bound on \Kj\. Combining (16) and (13), we obtain 

h(d 2 + 2rf, ) > 2 - h{da) - Ri - R 2 , (17) 

which is the second part of the theorem, the remainder following on 
interchanging the roles of /i and f 2 . 

In the next section, we put the epsilons into the above outline to 
give a precise proof. 

IV. PROOF OF THE THEOREM 

In order to make the ideas in the previous section precise, we must 
show that in some approximate sense equations (6) and (7) can be 
assumed to hold. To do this, we begin as follows. As in Section III, let 
(fu h, go, g\, gz) be a code with parameters (N, Mi, D , D u D 2 ), and 
let Ai, Bj, Cij be as defined in (5). Define the set S C 3& N as the set of 
x such that 

irf(x,ft«/.(x))<D„ + fi I a =1,2, (18a) 



and 



ilog|A, l(x ,| >!-*,-«, (18b) 



where S > is an arbitrary fixed parameter. Next, define the set / C 
{1, • ■ • , M\ } as the set of integers i such that 

P(A i nS)>8P(A i ). (19) 

Note that, if i e I, then AiHSj 6 <j>. Let x e A, n S, where i e I. Then 

-log|A f nS|-ilog|i4 /lW nS| 

£:^logfi + ilog|A, l(r) | 

>-log$+(l-.Ri-6). (20) 

Finally, paralleling (8), define 

K'j= X (AiHS), l<y<M 2 . (21) 

itI:Cyf)8i*4t 

If we imagine for now that P(S) and Pr{/i(X) e 1} are close to 1, 
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then K'j is an approximation of Kg. As in Section III, the theorem is 
proved by establishing two bounds on | K'j \ . 

4. 1 Upper bound on | Kj \ 

As in Section III, for 1 < j < M 2 , let Ay be the diameter of K'j, so 
that 

ilog|*;|</i(A;/2N). (22) 

Also, let xi, x 2 eK'j achieve d w (xi, x 2 ) = Aj. Hence a pair xi, x 2 exists 
such that 

A(xi)= /ito). A(x 2 ) = A(x 2 ) 

and 

/2<Xi)= /2(X 2 )=/ 

Also, xi , x 2 , xi , x 2 e S. Thus 

d fl (xi, xi), rf//(x 2 , x 2 ) < 2(Di + 8)N 

and 

<fo(xi, x 2 ) S 2(D 2 + 8)N, 

so that the triangle inequality yields 

£L < D 2 + 22), + 36. (23) 

2/V 

Inequalities (22) and (23) yield, on averaging over 7, 

1 Mi 

- £ Pr{ f»(x) = >}log \K'j I < /i(A + 2Z)i + 35) , (24) 

iV y =i 

which is our upper bound. 

4.2 Lower bound on |K / 1 

Let/' e {1, • • •, M2}, and let i be an integer in the summation set of 
(21), i.e., i e I, dj OS * <t>. Thus (20) yields 

-\og\A, n S\ >^\og 8 + (I - Ri - 8). 
Using this fact, we have from (21), for 1 <> < M 2 , 

i T iog|tf;i>iiogro; + 4i°gS + tt-fli- fi ). (25) 

N xV iV 

where m 7 ' 4 | [i e l:C t , ■ n S ^ <M | is the number of terms in the 
summation of (21). We must now lower bound mj. 
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Let ^(x), x e & N , be defined by 

1, xeS, /i(x)e/, 



^ (x) '0, otherwise. (26) 

Then from the rate-distortion bound (as in (15)), 

AT(1 - h(D )) < I(fi(X), / 2 (X); X) < H(f u f 2 , *) 

= tf(*)+#</2,|*)+#(/i|/2,*) 

= HW)+H(f 2 \4<) 

+ SPr{/2=>}PrW=l|/2=y}H(A|/ 2 =y,^=l) 

+ I Pr^ = 0}Pr{/ 2 =j\*= 0}/f(/i | /- 2 =y, t// = 0). 

Let it = Pr ty(x) = 0} . Then #ty) < h(rr). Also #( /" 2 | ^) < log M 2 and 
H(fi | / 2 =7, ^ = 0) < log Mi, thus, 

N(l - h(D )) < h{ir) + log M 2 + -n log Mi 

+ I Pr{/ 2 =7}PrW = 1 1 h =J)H(U \ f 2 =j, * = 1). 

Now suppose that x is such that / 2 (x) = j, »//(x) = 1. Let /i(x) = io. 
From (26), io e I, and from the definition (5c), Cij ^ <$>. Thus to is one 
of the m'j terms in the summation (21) defining K). We can therefore 
overbound H(fi \ f 2 = j, \p = 1) by log m'j, yielding 

N(l - h(D )) < h(ir) + log M 2 + tt log Mi 

M 2 

+ SPr{/ 2 =>}logm;. (27) 

Combining (27) with (25), we have 
1 m 2 
t; £ Pr{/ 2 =y}log | K'j 1 1 > 2 - ft(Z*>) - «i - « 

-ilogM 2 -^-^logM 1+ ilog«, (28) 

which is our lower bound. 
4.3 Conclusion of the proof 

Combining the bounds of (24) and (28), we have 

h(D 2 + 2D! + 35) > 2 - h(D Q ) -fa- — log M 2 

"^T" 5 "^ logMl + ^ log5 - (29) 

At the conclusion of this section, we prove the following lemma. 
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Lemma: If (Ri, R 2 , do, d\, d 2 ) is achievable, then for all 8, e > 0, 
there exists, for N sufficiently large, a code with parameters (N, Mi , 
M 2 , Do, D U D 2 ), where M„ <=2 ( *" +e,iv (a = 1, 2), D a <rf„ + e (a = 0, 1, 2), 
and Pr{\p(x) = 0} = -n < 8 + e. (Note that it depends on S and I, which 
in turn depend on 8.) 

Now, suppose that (JRi, R 2 , do, d u d 2 ) is achievable. Pick e, 8 > 
and let N be large enough so that the code described in the lemma 
exists. Applying (29) to this code, we have 

h(d 2 + 2di + 35 + 3e) > 2 - h(d + e) - Ri - (R 2 + e) 

h(8 + e) 1 

- 8- N - (8 + e)(R 2 + e) + -log 5. 

Letting N — > oo, and then 8, e — » 0, we have 

h(d 2 + 2d 1 ) > 2 - ft(cfo) - Ri - R 2 , 

which is the theorem. 
It remains to give a 
Proof of the Lemma: Let (xi, x 2 , • • ■) be an infinite binary sequence. 
Define x™ = (x n , • • • x m ), 1 <n<m< oo. Now let (/i, / 2 , £o, &,&) be 
a code with parameters (TV, Mi, M 2 , Do, D x , D 2 ). For K = 1, 2, • • •, 
define a "super" code which operates on K successive blocks of (Xi , 
X 2 , •••) in the obvious way. That is, letting q 1} q 2 be one-to-one 
mappings 

qi:{l,...,Mi) K ^{l, ...,Mf}, 

q 2 :{l,---,M 2 ) K ^{l,...,m), 

then the super code (/i, f 2 , go, gi, g 2 ) is the code with parameters 
(KN, M?, M?, Do,D u D 2 ) defined by 

fi{Xf N ) = qi[?i(X?), /,<Xj&), .-., A(X{£i,tf + i)] 
and 

£i(i) = [ii(ii), •••,ii(i/c)], 

where (i'i, • • •, i*) = qi 1 (i), 1 < i < Mf , etc. 

By the law of large numbers, for a = 1, 2, as K —> 00 (with N held 
fixed), 

1 
KN 



d(X? N , go o f a (X? N )) -^ D a , 



and 



^ d(x?*, go(Mxr), f 2 (x?*m -+ Do. 



Also, let #, = H(fi(X?)). For 8 > 0, define 
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f 1 #1 «1 

= (i:-log|A|>l- ¥ -^j. 

By the aep, as K -> oo 

Pr(/i(Xr)e/o}^l. 
Further, since Ai/ft < Ui, we have, as -KT — > oo, 

PrU-Iog|iW*>| > 1 -Hi - *}-> 1. 

We now take a look at the lemma. Let(Ri, ^2, do, di, cfe) be 
achievable. Let e > be given, and let ( /1 , f 2 , go , £1 , #2 ) be a code with 
parameters (AT, M,, M 2 , Do, D u D 2 ) such that M a <2 NlRa+t) , D a < d a 
+ e, a = 0, 1, 2. Construct the code (/i, h,go,g\*gz) with parameters 
(AT, M,, M 2 , A>, Di, Dz) where N - iiTiV, M a = M*, a = 1, 2, as above. 
Let S C a N be as defined by (18) with 5 > specified. Then, for K 
sufficiently large, we can make 

P(S C ) < e/2 

Then, with «//(.) defined by (24) and /by (19), 

7r = PrWXn = 0}=Pr{/ 1 (X)</ or XjS) 

<Pr{/-,(X)^/}+Pr{X^S} 

- £ P(Ai) + P(S C ) 

= l P(Ai n s c ) + £ ^(A* n s c ) + P(S C ) 

< ^ 5P(A,) + £ /'(A.- n s c ) + P(S C ) 

«v / v / 

< 5 + 2P(S C ) < e + 6\ 

which establishes the lemma. 
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